In view of the current efficiency problem of speaker recognition system, this paper utilized the tactics of integration algorithm to put forward a new kind of speaker recognition system framework. The traditional Maximum A posteriori Probability Vector Quantization (VQ-MAP) algorithm only focuses on the average vector regardless of weight. In order to solve this problem, this paper put forward an improved algorithm based on VQ-MAP. The algorithm used weighted average vector instead of average vector. Moreover, Support Vector Machine (SVM) algorithm costs too much time, so Least Squares Support Vector Machine (LS-SVM) was used instead of SVM. Finally, in the speaker recognition system, this paper used the parameters calculated from the improved VQ-MAP algorithm as training set of LS-SVM. The experimental results show that, the modeling time of integration algorithm based on improved VQ-MAP and LS-SVM is about 40% less than that of traditional SVM algorithm when using the Radial Basis Function (RBF) kernel function and the sample of 40 people. As the threshold value is 1 and the test speech time is 4 s, compared to the traditional VQ-MAP and SVM algorithm, the deterrent rate is reduced by 1.1%, the false rejection rate is reduced by 2.9% and the recognition rate is increased by 3.9%. As the threshold value is 1 and the test speech time is 4 s, compared to the traditional VQ-MAP and LS-SVM algorithm, the deterrent rate is reduced by 3.6%, the false rejection rate is reduced by 2.7% and the recognition rate is increased by 4.4%. The results show that the integrated algorithm can improve the recognition rate effectively and reduce the operation time significantly, meanwhile reduce the deterrent rate and the false rejection rate.